{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd \n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')\n",
    "plt.style.use('ggplot')\n",
    "plt.rcParams['figure.figsize'] = 10,7\n",
    "\n",
    "import sys\n",
    "sys.path.append('../src')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation\n",
    "\n",
    "- Download the data from: http://www.kibot.com/buy.aspx at the: \"Free historical data for your data quality analysis\" section\n",
    "- We have the data from the WDC stock and the iShares IVE ETF: https://www.ishares.com/us/products/239728/ishares-sp-500-value-etf "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tick Data info from kibot\n",
    "- http://www.kibot.com/support.aspx#data_format\n",
    "- The order of the fields in the tick files (with bid/ask prices) is: Date,Time,Price,Bid,Ask,Size. \n",
    "- The bid/ask prices are recorded whenever a trade occurs and they represent the \"national best bid and offer\" (NBBO) prices across multiple exchanges and ECNs.\n",
    "- For each trade, current best bid/ask values are recorded together with the transaction price and volume. Trade records are not aggregated and all transactions are included in their consecutive order.\n",
    "- The order of fields in our regular tick files (without bid/ask) is: Date,Time,Price,Size.\n",
    "- The order of fields in our 1, 5 or 10 second files is: Date,Time,Open,High,Low,Close,Volume. It is the same format used in our minute files.\n",
    "- The stocks and ETFs data includes pre-market (8:00-9:30 a.m. ET), regular (9:30 a.m.-4:00 p.m. ET.) and after market (4:00-6:30 p.m. ET) sessions.\n",
    "- Trading for SPY (SPDR S&P 500 ETF) and some other liquid ETFs and stocks usually starts at 4 a.m and ends at 8 p.m. ET."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_data_kibot(raw_data_path, out_data_path):\n",
    "    df = pd.read_csv(raw_data_path,\n",
    "                     header=None,\n",
    "                     names=['day', 'time', 'price', 'bid', 'ask', 'vol'])\n",
    "    df['date'] = pd.to_datetime(df['day'] + df['time'],\n",
    "                                format='%m/%d/%Y%H:%M:%S')\n",
    "    df['dollar_vol'] = df['price']*df['vol']\n",
    "    df = df.set_index('date')\n",
    "    df = df.drop(['day', 'time'],\n",
    "                 axis=1)\n",
    "    df = df.drop_duplicates()\n",
    "    df.to_parquet(out_data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data_path = '../data/raw/IVE_tickbidask_new.txt'\n",
    "out_data_path = '../data/interim/IVE_tickbidask.parq'\n",
    "prepare_data_kibot(raw_data_path, out_data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data_path = '../data/raw/WDC_tickbidask.txt'\n",
    "out_data_path = '../data/interim/WDC_tickbidask.parq'\n",
    "prepare_data_kibot(raw_data_path, out_data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Cleaning\n",
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#out_data_path = '../data/interim/WDC_tickbidask.parq'\n",
    "out_data_path = '../data/interim/IVE_tickbidask.parq'\n",
    "df = pd.read_parquet(out_data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAGfCAYAAABsocdzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFQpJREFUeJzt3V+M1PX97/HXsFuNQFhYxj/ZpU27FXOqUWuDkWgsFOei0TY/0guSWppQL0wlgcTERGJSb7zZXiAUK/GijTb1JE0vCk2NvdlQaAIhWUUTi63VxDSVtcFlFwJiVWDOhemeQw7Kuuy7w8DjceUM3+983zvJTp5+PsOXRrvdbgcAgFk1p9MDAABcikQWAEABkQUAUEBkAQAUEFkAAAVEFgBAAZEFAFBAZAEAFBBZAAAFRBYAQIHeTg/wH2NjY50eAbjENJvNjI+Pd3oM4BIzMDAwreOsZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAUEFkAAAVEFgBAAZEFAFBAZAEAFBBZAAAFRBYAQIHe8x2wffv2HDhwIH19fdm8eXOS5Ne//nVefvnl9Pb25tprr8369eszb968JMmOHTuya9euzJkzJz/60Y/y9a9/vfYnAAC4CJ03slauXJlvf/vbefrpp6eeu+WWW3L//fenp6cnzz//fHbs2JG1a9fmnXfeyb59+/Lkk09mcnIyTzzxRH72s59lzhwLZsBnGxwc7PQI03bo0KFOjwB0gfNG1o033pjDhw+f9dytt9469d833HBD9u/fnyQZHR3NnXfemS984Qu55pprct111+Wtt97KDTfcMMtjA5eainAZHBwUREDHXPAS065du6a2BCcmJrJ48eKpP+vv78/ExMSFXgIAoOucdyXrs/zud79LT09P7r777s997sjISEZGRpIkw8PDaTabFzIKwDn5bAE6ZcaRtXv37rz88st5/PHH02g0knyycnXkyJGpYyYmJtLf33/O81utVlqt1tTj8fHxmY4C8Kl8tgCzbWBgYFrHzWi78NVXX83vf//7PProo7nyyiunnl+2bFn27duXjz/+OIcPH867776b66+/fiaXAADoao12u93+rAO2bt2a119/PcePH09fX1/WrFmTHTt25NSpU5k/f36SZOnSpXnwwQeTfLKF+Kc//Slz5szJunXrctttt01rkLGxsQv8UQDO5ovvQIXprmSdN7L+W0QWMNtEFlChdLsQAIDPJrIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCgQG+nBwC6z0033ZSjR492eoxpGRwc7PQI57Vw4cIcPHiw02MAs0xkAZ/b0aNHc+jQoU6PcV7NZjPj4+OdHuO8uiEEgc/PdiEAQAGRBQBQ4Lzbhdu3b8+BAwfS19eXzZs3J0lOnDiRLVu25L333svVV1+dhx9+OPPnz0+73c6zzz6bV155JVdeeWXWr1+foaGh8h8CAOBic96VrJUrV+axxx4767mdO3fm5ptvzrZt23LzzTdn586dSZJXXnkl//rXv7Jt27Y8+OCD+cUvflEzNQDARe68kXXjjTdm/vz5Zz03OjqaFStWJElWrFiR0dHRJMlLL72Ub37zm2k0Grnhhhvy/vvvZ3JysmBsAICL24y+k3Xs2LEsWrQoySd/9fjYsWNJkomJiTSbzanjFi9enImJiVkYEwCgu1zwLRwajUYajcbnPm9kZCQjIyNJkuHh4bPiDLj4dcPvbG9vb1fMmXTH+wl8PjOKrL6+vkxOTmbRokWZnJzMggULkiT9/f1n3ZPmyJEj6e/vP+drtFqttFqtqcfdcC8b4P/qht/ZbrlPVtId7yfwiYGBgWkdN6PtwmXLlmXPnj1Jkj179uT222+fev7Pf/5z2u12/v73v2fu3LlT24oAAJeT865kbd26Na+//nqOHz+eH//4x1mzZk1Wr16dLVu2ZNeuXVO3cEiS2267LQcOHMjGjRtzxRVXZP369eU/AADAxajRbrfbnR4iScbGxjo9AjBNg4OD/lmdWdQt7yfwidLtQgAAPpvIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKNBot9vtTg+RJGNjY50eAZim//nff+v0CJec3//gf3V6BGCaBgYGpnWcyAI+t8HBwRw6dKjTY5xXs9nM+Ph4p8c4r255P4FPTDeybBcCABQQWQAABUQWAEABkQUAUEBkAQAUEFkAAAVEFgBAAZEFAFBAZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAU6L2Qk1944YXs2rUrjUYjX/ziF7N+/focPXo0W7duzfHjxzM0NJQNGzakt/eCLgMA0HVmvJI1MTGRP/7xjxkeHs7mzZtz5syZ7Nu3L88//3zuu+++PPXUU5k3b1527do1m/MCAHSFC9ouPHPmTD766KOcPn06H330URYuXJiDBw9m+fLlSZKVK1dmdHR0VgYFAOgmM97H6+/vz3e/+9089NBDueKKK3LrrbdmaGgoc+fOTU9Pz9QxExMT5zx/ZGQkIyMjSZLh4eE0m82ZjgJ0QDf8zvb29nbFnEl3vJ/A5zPjyDpx4kRGR0fz9NNPZ+7cuXnyySfz6quvTvv8VquVVqs19Xh8fHymowAd0A2/s81msyvmTLrj/QQ+MTAwMK3jZhxZr732Wq655posWLAgSXLHHXfkjTfeyMmTJ3P69On09PRkYmIi/f39M70EAEDXmvF3sprNZt588818+OGHabfbee2117JkyZLcdNNN2b9/f5Jk9+7dWbZs2awNCwDQLWa8krV06dIsX748jz76aHp6evLlL385rVYr3/jGN7J169b85je/yVe+8pWsWrVqNucFAOgKjXa73e70EEkyNjbW6RGAaRocHMyhQ4c6PcZ5dct3srrl/QQ+Md3vZLnjOwBAAZEFAFBAZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAUEFkAAAVEFgBAAZEFAFBAZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAUEFkAAAVEFgBAAZEFAFBAZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAUEFkAAAVEFgBAAZEFAFBAZAEAFOjt9ABAdxocHOz0CJeMhQsXdnoEoIDIAj63Q4cOdXqEaRkcHOyaWYFLj+1CAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKHBB98l6//3388wzz+Sf//xnGo1GHnrooQwMDGTLli157733cvXVV+fhhx/O/PnzZ2teAICu0Gi32+2Znvzzn/88X/va13LPPffk1KlT+fDDD7Njx47Mnz8/q1evzs6dO3PixImsXbv2vK81NjY20zEAzsnNSIEKAwMD0zpuxtuFJ0+ezF//+tesWrUqSdLb25t58+ZldHQ0K1asSJKsWLEio6OjM70EAEDXmvF24eHDh7NgwYJs3749//jHPzI0NJR169bl2LFjWbRoUZJP/j2uY8eOnfP8kZGRjIyMJEmGh4fTbDZnOgrAp/LZAnTKjCPr9OnTefvtt/PAAw9k6dKlefbZZ7Nz586zjmk0Gmk0Guc8v9VqpdVqTT0eHx+f6SgAn8pnCzDbyrcLFy9enMWLF2fp0qVJkuXLl+ftt99OX19fJicnkySTk5NZsGDBTC8BANC1ZhxZCxcuzOLFi6e+sP7aa69lyZIlWbZsWfbs2ZMk2bNnT26//fbZmRQAoItc0C0cHnjggWzbti2nTp3KNddck/Xr16fdbmfLli3ZtWvX1C0cAAAuNxd0C4fZ5BYOwGxzCwegQvl3sgAA+HQiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAK9F7oC5w5cyabNm1Kf39/Nm3alMOHD2fr1q05fvx4hoaGsmHDhvT2XvBlAAC6ygWvZL344osZHBycevz888/nvvvuy1NPPZV58+Zl165dF3oJAICuc0GRdeTIkRw4cCD33HNPkqTdbufgwYNZvnx5kmTlypUZHR298CkBALrMBe3jPffcc1m7dm0++OCDJMnx48czd+7c9PT0JEn6+/szMTFxznNHRkYyMjKSJBkeHk6z2byQUQDOyWcL0CkzjqyXX345fX19GRoaysGDBz/3+a1WK61Wa+rx+Pj4TEcB+FQ+W4DZNjAwMK3jZhxZb7zxRl566aW88sor+eijj/LBBx/kueeey8mTJ3P69On09PRkYmIi/f39M70EAEDXmnFk3X///bn//vuTJAcPHswf/vCHbNy4MU8++WT279+fu+66K7t3786yZctmbVgAgG4x6/fJ+sEPfpAXXnghGzZsyIkTJ7Jq1arZvgQAwEWv0W63250eIknGxsY6PQJwiRkcHMyhQ4c6PQZwiZnud7Lc8R0AoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoEBvpwcASJLBwcGued1Dhw7N+msClx6RBVwUKsKl2WxmfHx81l8XYDpsFwIAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAUEFkAAAVEFgBAAZEFAFBAZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAU6J3piePj43n66adz9OjRNBqNtFqt3HvvvTlx4kS2bNmS9957L1dffXUefvjhzJ8/fzZnBgC46M04snp6evLDH/4wQ0ND+eCDD7Jp06bccsst2b17d26++easXr06O3fuzM6dO7N27drZnBkA4KI34+3CRYsWZWhoKEly1VVXZXBwMBMTExkdHc2KFSuSJCtWrMjo6OjsTAoA0EVm5TtZhw8fzttvv53rr78+x44dy6JFi5IkCxcuzLFjx2bjEgAAXWXG24X/8e9//zubN2/OunXrMnfu3LP+rNFopNFonPO8kZGRjIyMJEmGh4fTbDYvdBSAs/T29vpsATrmgiLr1KlT2bx5c+6+++7ccccdSZK+vr5MTk5m0aJFmZyczIIFC855bqvVSqvVmno8Pj5+IaMA/H+azabPFmDWDQwMTOu4GW8XttvtPPPMMxkcHMx3vvOdqeeXLVuWPXv2JEn27NmT22+/faaXAADoWo12u92eyYl/+9vf8vjjj+dLX/rS1Jbg97///SxdujRbtmzJ+Pj457qFw9jY2EzGAPhUVrKACtNdyZpxZM02kQXMNpEFVCjfLgQA4NOJLACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACvZ0eAGC2LVmyJO12e+pxo9HIO++808GJgMuRlSzgkvKfwJo7d27279+fuXPnpt1uZ8mSJZ0eDbjMlK1kvfrqq3n22Wdz5syZ3HPPPVm9enXVpQCm/Cew3nzzzTSbzbz55ptZunRpTp482enRgMtMyUrWmTNn8stf/jKPPfZYtmzZkr1791qqB/5rNmzYkFWrVuWqq67KqlWrsmHDhk6PBFyGSlay3nrrrVx33XW59tprkyR33nlnRkdHLdcD/xU//elP89vf/jb33ntvXnzxxaxZs6bTIwGXoZKVrImJiSxevHjq8eLFizMxMVFxKYBzWrduXf7yl79k3bp1nR4FuEx17G8XjoyMZGRkJEkyPDycZrPZqVGAS8icOZ/8v+PJkyezfPnys57zOQP8N5VEVn9/f44cOTL1+MiRI+nv7z/rmFarlVarNfV4fHy8YhTgMrN06dI88cQTueuuu9JsNjM+Pp69e/fmJz/5ic8ZYFYMDAxM67iS7cKvfvWreffdd3P48OGcOnUq+/bty7JlyyouBXCWjRs35pFHHsnevXvz8ccfZ+/evXnkkUeycePGTo8GXGYa7f/3jn2z6MCBA/nVr36VM2fO5Fvf+la+973vfebxY2NjFWMAl6GdO3dm27ZtU7dv2Lhxo9vIALNmuitZZZH1eYksYLb9Z7sQYDZ1dLsQAOByJ7IAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoMBF8w9EAwBcSqxkAZesTZs2dXoE4DImsgAACogsAIACIgu4ZLVarU6PAFzGfPEdAKCAlSwAgAK9nR4AYLZt3749Bw4cSF9fXzZv3tzpcYDLlJUs4JKzcuXKPPbYY50eA7jMiSzgknPjjTdm/vz5nR4DuMyJLACAAiILAKCAyAIAKCCyAAAKuBkpcMnZunVrXn/99Rw/fjx9fX1Zs2ZNVq1a1emxgMuMyAIAKGC7EACggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAv8H+oB1mgO7WlQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x504 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "_ = ax.boxplot(df.price)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.clean_data import mad_outlier\n",
    "mad = mad_outlier(df.price.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n"
     ]
    }
   ],
   "source": [
    "print(mad.sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "### drop the incorrect rows\n",
    "df = df.loc[~mad]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAGfCAYAAABsocdzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEaRJREFUeJzt3UGI1Pfdx/HPuNuk6OLqOrFlJ4V2qeGpaUhTDJWWVjFzC+XJyUPqIc0hNIKBnCI59FIoezFuTCM9FBKoD/RUDQ3tZdlqDyGwUVNE+6QtlNLutph1VdYYYtV5DvJsn4DJrrv7fSazeb1uM/7/8//OwA5v/r///G10Op1OAABYUWu6PQAAwGoksgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAK9Hd7gP81PT3d7RGAVabZbGZmZqbbYwCrzPDw8KK2cyYLAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAL9C21w+PDhnDp1KoODgzlw4ECS5Oc//3lOnjyZ/v7+fO5zn8vevXuzbt26JMnRo0czMTGRNWvW5Pvf/36+9rWv1b4DYFVotVrdHmHRpqamuj0C0AManU6n83EbnDt3Lp/97Gfz8ssvz0fW73//+3z1q19NX19fjhw5kiTZs2dP/v73v+fFF1/Mj3/841y8eDE/+tGP8uKLL2bNmoVPmE1PT6/A2wH4t1arJYiAFTc8PLyo7Rasn61bt2ZgYOBDzz344IPp6+tLktx3332ZnZ1NkkxOTuab3/xmPvOZz2Tz5s35/Oc/nz//+c93OjsAQM9b9jVZExMT80uCs7Oz2bRp0/y/DQ0NzQcYAMCnyYLXZH2cX/7yl+nr68u3v/3tO953fHw84+PjSZLR0dE0m83ljAJwW75bgG5ZcmQdP348J0+ezA9/+MM0Go0kt85cXbhwYX6b2dnZDA0N3Xb/druddrs9/3hmZmapowB8JN8twEpbsWuybuftt9/Oa6+9lueeey533333/PPbtm3LG2+8kX/96185f/58/vGPf+TLX/7yUg4BANDTFvx14djYWM6dO5e5ubkMDg5m9+7dOXr0aK5fvz5/QfyWLVvy1FNPJbm1hPjb3/42a9asyRNPPJGHHnpoUYP4dSGw0vy6EKiw2DNZC0bW/xeRBaw0kQVUKF0uBADg44ksAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACjQ3+0BgN5z//3359KlS90eY1FarVa3R1jQhg0bcvbs2W6PAawwkQXcsUuXLmVqaqrbYyyo2WxmZmam22MsqBdCELhzlgsBAAqILACAAiILAKCAyAIAKCCyAAAKLPjrwsOHD+fUqVMZHBzMgQMHkiRXrlzJwYMH8+677+aee+7Js88+m4GBgXQ6nbzyyis5ffp07r777uzduzcjIyPlbwIA4JNmwTNZO3fuzPPPP/+h544dO5YHHngghw4dygMPPJBjx44lSU6fPp1//vOfOXToUJ566qn87Gc/q5kaAOATbsHI2rp1awYGBj703OTkZHbs2JEk2bFjRyYnJ5Mkb731Vr7zne+k0Wjkvvvuy3vvvZeLFy8WjA0A8Mm2pJuRXr58ORs3bkxy607Fly9fTpLMzs6m2WzOb7dp06bMzs7Ob/t/jY+PZ3x8PEkyOjr6of2AT75e+Jvt7+/viTmT3vg8gTuz7Du+NxqNNBqNO96v3W6n3W7PP+6FuzID/9YLf7O9csf3pDc+T+CW4eHhRW23pF8XDg4Ozi8DXrx4MevXr0+SDA0NfeiL4sKFCxkaGlrKIQAAetqSImvbtm05ceJEkuTEiRN5+OGH55//3e9+l06nkz/+8Y9Zu3btbZcKAQBWuwWXC8fGxnLu3LnMzc3lBz/4QXbv3p3HHnssBw8ezMTExPwtHJLkoYceyqlTp/LMM8/krrvuyt69e8vfAADAJ1Gj0+l0uj1EkkxPT3d7BGCRWq1Wpqamuj3Ggnrlmqxe+TyBW0qvyQIA4OOJLACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKNDqdTqfbQyTJ9PR0t0cAFuk//+u/uz3CqvPa9/6j2yMAizQ8PLyo7UQWcMdarVampqa6PcaCms1mZmZmuj3Ggnrl8wRuWWxkWS4EACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAL9y9n59ddfz8TERBqNRr7whS9k7969uXTpUsbGxjI3N5eRkZHs27cv/f3LOgwAQM9Z8pms2dnZ/OY3v8no6GgOHDiQmzdv5o033siRI0fy6KOP5qWXXsq6desyMTGxkvMCAPSEZS0X3rx5M9euXcuNGzdy7dq1bNiwIWfPns327duTJDt37szk5OSKDAoA0EuWvI43NDSU7373u3n66adz11135cEHH8zIyEjWrl2bvr6++W1mZ2dvu//4+HjGx8eTJKOjo2k2m0sdBeiCXvib7e/v74k5k974PIE7s+TIunLlSiYnJ/Pyyy9n7dq1eeGFF/L2228vev92u512uz3/eGZmZqmjAF3QC3+zzWazJ+ZMeuPzBG4ZHh5e1HZLjqwzZ85k8+bNWb9+fZLkG9/4Rt55551cvXo1N27cSF9fX2ZnZzM0NLTUQwAA9KwlX5PVbDbzpz/9KR988EE6nU7OnDmTe++9N/fff3/efPPNJMnx48ezbdu2FRsWAKBXLPlM1pYtW7J9+/Y899xz6evryxe/+MW02+18/etfz9jYWH7xi1/kS1/6Unbt2rWS8wIA9IRGp9PpdHuIJJmenu72CMAitVqtTE1NdXuMBfXKNVm98nkCtyz2mix3fAcAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACvR3ewCgN7VarW6PsGps2LCh2yMABUQWcMempqa6PcKitFqtnpkVWH0sFwIAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAUEFkAAAWW9d/qvPfee/npT3+av/3tb2k0Gnn66aczPDycgwcP5t13380999yTZ599NgMDAys1LwBAT2h0Op3OUnf+yU9+kq985St55JFHcv369XzwwQc5evRoBgYG8thjj+XYsWO5cuVK9uzZs+BrTU9PL3UMgNvyfxcCFYaHhxe13ZKXC69evZo//OEP2bVrV5Kkv78/69aty+TkZHbs2JEk2bFjRyYnJ5d6CACAnrXk5cLz589n/fr1OXz4cP76179mZGQkTzzxRC5fvpyNGzcmSTZs2JDLly+v2LAAAL1iyZF148aN/OUvf8mTTz6ZLVu25JVXXsmxY8c+tE2j0Uij0bjt/uPj4xkfH0+SjI6OptlsLnUUgI/kuwXoliVH1qZNm7Jp06Zs2bIlSbJ9+/YcO3Ysg4ODuXjxYjZu3JiLFy9m/fr1t92/3W6n3W7PP56ZmVnqKAAfyXcLsNLKr8nasGFDNm3aNH/B+pkzZ3Lvvfdm27ZtOXHiRJLkxIkTefjhh5d6CACAnrWsWzg8+eSTOXToUK5fv57Nmzdn79696XQ6OXjwYCYmJuZv4QAA8GmzrFs4rCS3cABWmls4ABXKlwsBAPhoIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAiILAKCAyAIAKNC/3Be4efNm9u/fn6Ghoezfvz/nz5/P2NhY5ubmMjIykn379qW/f9mHAQDoKcs+k/XrX/86rVZr/vGRI0fy6KOP5qWXXsq6desyMTGx3EMAAPScZUXWhQsXcurUqTzyyCNJkk6nk7Nnz2b79u1Jkp07d2ZycnL5UwIA9JhlreO9+uqr2bNnT95///0kydzcXNauXZu+vr4kydDQUGZnZ2+77/j4eMbHx5Mko6OjaTabyxkF4LZ8twDdsuTIOnnyZAYHBzMyMpKzZ8/e8f7tdjvtdnv+8czMzFJHAfhIvluAlTY8PLyo7ZYcWe+8807eeuutnD59OteuXcv777+fV199NVevXs2NGzfS19eX2dnZDA0NLfUQAAA9a8mR9fjjj+fxxx9Pkpw9eza/+tWv8swzz+SFF17Im2++mW9961s5fvx4tm3btmLDAgD0ihW/T9b3vve9vP7669m3b1+uXLmSXbt2rfQhAAA+8RqdTqfT7SGSZHp6utsjAKtMq9XK1NRUt8cAVpnFXpPlju8AAAVEFgBAAZEFAFBAZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAUEFkAAAVEFgBAAZEFAFBAZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUEBkAQAUEFkAAAVEFgBAAZEFAFBAZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABUQWAEABkQUAUKC/2wMAJEmr1eqZ152amlrx1wRWH5EFfCJUhEuz2czMzMyKvy7AYlguBAAoILIAAAqILACAAiILAKCAyAIAKCCyAAAKiCwAgAIiCwCggMgCACggsgAACogsAIACIgsAoIDIAgAoILIAAAqILACAAo1Op9Pp9hAAAKuNM1nAqrV///5ujwB8ioksAIACIgsAoIDIAlatdrvd7RGATzEXvgMAFHAmCwCgQH+3BwBYaYcPH86pU6cyODiYAwcOdHsc4FPKmSxg1dm5c2eef/75bo8BfMqJLGDV2bp1awYGBro9BvApJ7IAAAqILACAAiILAKCAyAIAKOBmpMCqMzY2lnPnzmVubi6Dg4PZvXt3du3a1e2xgE8ZkQUAUMByIQBAAZEFAFBAZAEAFBBZAAAFRBYAQAGRBQBQQGQBABQQWQAABf4HXbtKu8MvKccAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x504 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "_ = ax.boxplot(df.price)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean_data_path = '../data/processed/clean_WDC_tickbidask.parq'\n",
    "clean_data_path = '../data/processed/clean_IVE_tickbidask.parq'\n",
    "df.to_parquet(clean_data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test clean data loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_data_path = '../data/processed/WDC_tickbidask.parq'\n",
    "clean_data_path = '../data/processed/clean_IVE_tickbidask.parq'\n",
    "df = pd.read_parquet(clean_data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>bid</th>\n",
       "      <th>ask</th>\n",
       "      <th>vol</th>\n",
       "      <th>dollar_vol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2009-09-28 09:30:00</th>\n",
       "      <td>50.79</td>\n",
       "      <td>50.70</td>\n",
       "      <td>50.79</td>\n",
       "      <td>100</td>\n",
       "      <td>5079.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-09-28 09:30:00</th>\n",
       "      <td>50.71</td>\n",
       "      <td>50.70</td>\n",
       "      <td>50.79</td>\n",
       "      <td>638</td>\n",
       "      <td>32352.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-09-28 09:31:32</th>\n",
       "      <td>50.75</td>\n",
       "      <td>50.75</td>\n",
       "      <td>50.76</td>\n",
       "      <td>100</td>\n",
       "      <td>5075.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-09-28 09:31:33</th>\n",
       "      <td>50.75</td>\n",
       "      <td>50.72</td>\n",
       "      <td>50.75</td>\n",
       "      <td>100</td>\n",
       "      <td>5075.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-09-28 09:31:50</th>\n",
       "      <td>50.75</td>\n",
       "      <td>50.73</td>\n",
       "      <td>50.76</td>\n",
       "      <td>300</td>\n",
       "      <td>15225.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     price    bid    ask  vol  dollar_vol\n",
       "date                                                     \n",
       "2009-09-28 09:30:00  50.79  50.70  50.79  100     5079.00\n",
       "2009-09-28 09:30:00  50.71  50.70  50.79  638    32352.98\n",
       "2009-09-28 09:31:32  50.75  50.75  50.76  100     5075.00\n",
       "2009-09-28 09:31:33  50.75  50.72  50.75  100     5075.00\n",
       "2009-09-28 09:31:50  50.75  50.73  50.76  300    15225.00"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
